{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "W13_Tutorial",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W13_FutureDL/student/W13_Tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0jMtXo2lrqU"
      },
      "source": [
        "# CIS-522 Week 13\n",
        "# The Future of Deep Learning\n",
        "\n",
        "__Instructor__: Lyle Ungar\n",
        "\n",
        "__Content creators:__ Michael Zhou\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ymkujLBlqM6F"
      },
      "source": [
        "## Tutorial Objectives\n",
        "\n",
        "1. Understand where deep learning currently succeeds and where it fails \\\\\n",
        "2. Know the key invariances used in deep learning \\\\\n",
        "3. Appreciate the limits of symbolic and non-symbolic (discrete and continuous) reasoning \\\\\n",
        "4. Recognize where correlation and causality diverge and why causality matters \\\\\n",
        "5. See the promise of self-supervision, continuous learning, and curiosity \\\\"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F0zdndLoLDwY",
        "cellView": "form"
      },
      "source": [
        "# @title Week 13 Slides\n",
        "from IPython.display import HTML\n",
        "HTML('<iframe src=\"https://docs.google.com/presentation/d/1_OInMCvVhJjradtLAhq2GQ6XpKmISkwoo5h34DeR6_E/edit?usp=sharing\" frameborder=\"0\" width=\"960\" height=\"569\" allowfullscreen=\"true\" mozallowfullscreen=\"true\" webkitallowfullscreen=\"true\"></iframe>')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ULqtMpvDtxl4",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Intro\n",
        "import time\n",
        "try: t0;\n",
        "except NameError: t0=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"sPrwv5xiERU\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "a5NQBmTeSs5S"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "my_pennkey = '' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0LMCVKg3EOSn"
      },
      "source": [
        "## Recap the experience from the course\n",
        "\n",
        "What did you learn in the course? [15 min discussion]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0q2tL2Q5CAde",
        "cellView": "form"
      },
      "source": [
        "learning_from_the_course= '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_ZIAAg0ckH-"
      },
      "source": [
        "*Estimated time: 20 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqQJVzPYjRAI"
      },
      "source": [
        "---\n",
        "# Section 1: DL Successes and Limits"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oCE24_HD91S",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Successes and Limits\n",
        "\n",
        "try: t1;\n",
        "except NameError: t1=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"uhYDnJss2Fk\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BAkagV3mo7N"
      },
      "source": [
        "## Section 1.1: Successes in Applications"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwtavl6Base-"
      },
      "source": [
        "As a whole, deep learning models and techniques have a variety of useful applications for different industries and disciplines. Recently, much interest has been put into many focus areas of deep learning from molecular discovery and pharmaceuticals to autonomous vehicles and education technology. Other focus areas such as machine translation, fraud prevention, cosmetics, and e-sports have experienced tremendous growth in private investments by large tech companies in just the last few years alone.\n",
        "\n",
        "Deep learning models have found lots of potential use in certain types of applications. For instance, deep generative model architectures such as GANs have found use as tools for building complex 3-D simulations for the purposes of molecular dynamics and other scientific fields. Meanwhile, CNN-based models have been widely adopted for object recognition in real-world settings and images and transformers have been extensively used for NLP tasks such as machine translation and speech recognition."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQpRRSVenNLo"
      },
      "source": [
        "## Section 1.2: Limitations and Drawbacks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eLMK8jN4lAzd"
      },
      "source": [
        "For all the advantages and widespread usage of deep learning in many focus areas, there are certain real-world applications for which existing deep learning architectures and techniques are poorly equipped to handle.\n",
        "\n",
        "In vision applications, while deep learning models perform relatively well on image recognition tasks, even cutting-edge models often struggle with scene recognition tasks. Such tasks involve identifying entire environments or settings as well as sets of objects in these scenes and answering questions about these objects in relation to their respective scenes. \n",
        "\n",
        "In machine translation, transformer model architectures do generally well with translating simple sentences and phrases across languages. However, there are cases where such models struggle with the nuances of natural language understanding. Such failures center on factors like the inability to recognize synonyms, and the effects of built-in bias derived from the statistics of a language that may translate over well to another language that differs in key characteristics like grammar rules and pronoun usage. \n",
        "\n",
        "Scene captioning, which involves taking an image and deriving an appropriate caption describing what is going on from its deep representation in a neural network, is another area where current machine translation models usually struggle with. While such models generally do a good job describing what is in an image, most fail to demonstrate an understanding of what is going on in the image. Labels provided by such models end up being superficial, only focused on surface details while ignoring the main idea."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SAqGi3uS5tey"
      },
      "source": [
        "What are some other limitations for deep learning models that you have encountered? For what reasons do these models have such limits?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "v3rowQ3h5tez"
      },
      "source": [
        "deep_learning_limits = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-LmnyIzR6Hvh"
      },
      "source": [
        "## Section 1.3: One-Second Rule"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0QN4c_Vv6Kxs"
      },
      "source": [
        "For many potential deep learning applications, a good rule of thumb to follow is the \"one-second rule\". Generally, most of the tasks for which deep learning models perform well on are trivial tasks that take at most a few seconds. Such tasks include the recognition of a single object or person, the generation of simple image captions, simple sentence translation, and choosing an action in a game environment. These tasks are often characterized by quick reactions and short duration, with no need for deeper levels of understanding or comprehension."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T8sCdEyj5d4f"
      },
      "source": [
        "How accurate do you think the “one-second rule” is? Why do you think it tends to hold?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "HTypw50s5Q6K"
      },
      "source": [
        "one_second_rule = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVOk4X8d7eEB"
      },
      "source": [
        "One of the more interesting areas of research with regards to deep learning models concerns how to make them emulate human reasoning, which is defined by an ability to generalize well across different situations even without prior knowledge or exposure through abstract reasoning and concepts. Most current deep learning models struggle greatly in this respect, so we will take the time to explore some techniques that could make these models better prepared to handle such scenarios."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sjHNbw79hcHT"
      },
      "source": [
        "*Estimated time: 45 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Y1z2u3zjTxB"
      },
      "source": [
        "---\n",
        "# Section 2: Invariances"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LznDcyfhEY9w",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Invariances\n",
        "try: t2;\n",
        "except NameError: t2=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"6x0pC0ilZL8\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NubY-mCqoMmC"
      },
      "source": [
        "## Section 2.1: DL Invariances"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5vB5h-o_nFi"
      },
      "source": [
        "Increasingly, we have incorporated different types of invariances into deep learning model architectures to help them generalize across different situations and make them more scalable. Here are some examples of invariances used by deep learning models:\n",
        "\n",
        "*   **CNNs:** local feature detectors that assign higher correlation to pixels close in proximity (translational invariance)\n",
        "*   **RNNs**: local feature detectors that assign higher importance to events of the recent past over distant past (translational invariance)\n",
        "*   Laws of physics serving as parameter constraints (e.g. conservation of mass and energy)\n",
        "\n",
        "Pretraining also requires consistencies for similar types of inputs across domains. In the domains of vision and NLP, local feature detectors are expected to be relevant for different images or texts even if a model has not encountered them in training due to the assumption that future input images or texts will likely share similar features with the training examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Fl7v_8yF9sP"
      },
      "source": [
        "List five examples of invariances used in deep learning:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "W_auK1ODF9sh"
      },
      "source": [
        "invariance_examples = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XgEomACNoiTk"
      },
      "source": [
        "## Section 2.2: Capsules"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtViSaFFEw0i"
      },
      "source": [
        "Another type of construct often used in deep learning are capsules. Capsules mainly rely on equivariance, which is a property which results in the output undergoing an identical transformation to any applied to the input. This is in contrast to invariance, which results in no changes to output even with transformations in input. For images, equivariance means that any change in viewpoint of an image will result in nonlinear changes to pixels and linear changes to observed sub-objects (e.g. facial features like the nose, mouth, and eyes).\n",
        "\n",
        "Using equivariance, capsules learn to recognize individual sub-objects over a limited set of viewing conditions and image deformations. Deep learning models often use series of capsules, each acting as a module focusing on a specific sub-object of an entity. Each capsule outputs the probability that its associated sub-object is present in the image. It also outputs information about the sub-object's characteristics (e.g. pose, lighting, orientation), providing a canonical representation of the sub-object.  \n",
        "\n",
        "Taken together, these capsules are analogous to the convolution filters of CNNs in function, but are smarter and more complex in the extraction of information from features. Compared to simple filters, capsules work in series to incorporate multiple sources of quasi-independent or independent information together to derive a more sophisticated understanding of what they represent as a whole."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtVS1z1VhhMH"
      },
      "source": [
        "*Estimated time: 65 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIhJPnGqjUJs"
      },
      "source": [
        "---\n",
        "# Section 3: Discrete vs. Continuous"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IMTUSfDYEhYx",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Discrete vs Continuous\n",
        "\n",
        "try: t3;\n",
        "except NameError: t3=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"mtaNvwl5CiA\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uGO5-zhapGH0"
      },
      "source": [
        "## Section 3.1: Continuous Modeling and Discrete Optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-cE7PFmmiNXT"
      },
      "source": [
        "As we have observed with neural networks and similar models, one of the most central features of deep learning is the use of differentiable equations and gradient descent to approximate the world with discrete elements. \n",
        "\n",
        "Neural networks are used to approximate discrete functions to perform a variety of tasks in deep learning, from vision to NLP applications. Conceptually, discrete optimization is NP-hard. With the use of gradient descent, neural networks are able to derive approximations to most of these discrete problems. However, since gradient descent as a technique can only provide at best imperfect approximations, there are some problems that cannot be reliably solved with such methods.\n",
        "\n",
        "As a result of these shortcomings, much research has focused on continuous modeling methods. Such approaches rely on essentially gluing in discrete, symbolic representations of the world. They aim to emulate the human brain's ability to approximate discrete functions using purely continuous neural units, allowing for sparsity at the conceptual level.\n",
        "\n",
        "One of the most recent attempts at building a continuous model uses CLEVRER, a system that generates synthetic movies of different objects in motion relative to each other. These movies are then converted into sequences of images that are shown to a model, which is given a set of conceptual questions to answer based on the image sequence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5ljDZCgGeB9"
      },
      "source": [
        "Machine translation was, until fairly recently, done using parsing with large numbers of (probabilistic) rules. It is now virtually all done using transformers. Why do you think deep learning might be more accurate than a rule-based system?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "vXyaKc1TGeCL"
      },
      "source": [
        "why_no_rules = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_4YH_dtnpNIj"
      },
      "source": [
        "## Section 3.2: Neurosymbolic Reasoning and Programming"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VhAFlrIWsol1"
      },
      "source": [
        "Conventional neural networks are able to answer certain types of questions posed by systems like CLEVRER reasonably well. However, there are other types of questions which neural networks are not as well equipped to handle reliably, such as counterfactual questions. To help models answer these kinds of questions, one can incorporate neurosymbolic reasoning into their architecture.\n",
        "\n",
        "Neurosymbolic reasoning focuses on having a model learn mappings from images to descriptions via a discrete\n",
        "representation of objects with features and relationships. Essentially, the main idea is to form a deep learning model able to map an image input to a discrete representation that is learned from attributes of objects and the relations between them rather than a representation built into the architecture.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_IfNXn1Gegs"
      },
      "source": [
        "What might neuro-symbolic systems be able to learn with less data and generalize better? Discuss in the context of an AI system that takes in a list of parts for building a device (nuts, bolts, motors, servos, batteries, etc.) and tells you what it might be missing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "_RP_-yFgGegt"
      },
      "source": [
        "neurosymbolic_learning = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aF1Xa6LFhmVH"
      },
      "source": [
        "*Estimated time: 85 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mNMG4v49lBCZ"
      },
      "source": [
        "---\n",
        "# Section 4: Importance of Causality"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4QJmbUrtICWJ",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Correlation\n",
        "\n",
        "try: t4;\n",
        "except NameError: t4=time.time()\n",
        "\n",
        "from IPython.display import YouTubeVideo\n",
        "video = YouTubeVideo(id=\"7PGbmQYqxMg\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nkttl-dQpu1p"
      },
      "source": [
        "## Section 4.1: Generalization and Taking Actions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QqYAhZT02Au_"
      },
      "source": [
        "From our discussion on building continuous models able to handle counterfactual questions (i.e. \"What would happen if ...\"?) rather than just simply making predictions, we began to touch upon the idea of causality. \n",
        "\n",
        "In the context of deep learning, causality is a desirable trait that allows models to generalize better. In other words, causal models are not as susceptible to changes in dataset correlations as they are able to detect underlying fundamental factors or patterns. Causal models also support action selection, allowing models to answer hypothetical questions with more certainty.\n",
        "\n",
        "It is important to note correlation is not the same as causality in most real-world situations. Beyond reinforcement learning, which deals with correlation and causality through the learning of actions, most deep learning models find it difficult to derive causality from observations alone. Generally, the more modular a system is, the more likely it is to capture causality."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EFUySv6oG1A4"
      },
      "source": [
        "Why are modular systems more likely to capture causality?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "u6M3cCG9G1A4"
      },
      "source": [
        "modular_causality = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8A5pZiTqK1s"
      },
      "source": [
        "## Section 4.2: Recurrent Independent Mechanisms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXNADJReJlbY"
      },
      "source": [
        "We have gotten a glimpse at how causal models work in the real world as well as the challenges associated with creating them. One final note we should consider is that causal models often reside in the discrete space. This means they are defined by symbolic representations, dealing with physical objects rather than image objects. To effectively learn causal models which can generalize well without relatively little training data, we would need to move away toward this sparse, symbolic space.\n",
        "\n",
        "One approach that aims to emulate this process is Recurrent Independent Mechanisms, which focuses on embedding modularity into deep learning systems. Given a neural network that takes in a sequence of visual observations about the world and generates a set of hidden states, this approach will periodically initiate sparse communication via an attention mechanism. This attention mechanism is configured such that every time it activates, only a small number of hidden state modules are passed to the input for the next time step. \n",
        "\n",
        "The overall approach behaves as a particular type of recurrent neural network where in place of full connectivity, a degree of sparsity is embedded into the architecture. This sparsity is what allows the resulting model to generalize better than non-causal models. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jezBNEMJHAUs"
      },
      "source": [
        "What is the likely “signature” of a simulation for CLEVRER? What does it and does it not produce?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "MqWmLkQNHAUs"
      },
      "source": [
        "simulation_signature = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Pir1pyDGyvh"
      },
      "source": [
        "Why does the CLEVRER system build a model and run simulations instead of using the more standard deep learning structure mapping from observation to label or to future observation or from observation to action?  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "i1LHyB8sGyvi"
      },
      "source": [
        "learning_simulations = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PperXnBihq5d"
      },
      "source": [
        "*Estimated time: 105 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VqnK4-3plBYc"
      },
      "source": [
        "---\n",
        "# Section 5: Self-Supervision, Continual Learning, and Curiosity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WwR-vKoRIr5f",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Self Supervision\n",
        "\n",
        "try: t5;\n",
        "except NameError: t5=time.time()\n",
        "\n",
        "video = YouTubeVideo(id=\"83xFxdzBMb8\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVd5-BCYqnvF"
      },
      "source": [
        "## Section 5.1: Self-Supervision"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmk9gYOZKquL"
      },
      "source": [
        "We have seen the use of invariances and integration of causality as strategies we can use to make better and smarter deep learning models for more generalized situations. Three additional methods can be considered for going beyond the supervised learning paradigm that defines most deep learning models: self-supervision, continual learning, and curiosity.\n",
        "\n",
        "We have seen self-supervision as a critical component for certain types of models used to learn feature detectors which are subsequently fine tuned with machine learning. For instance, self-supervision in autoencoders allows them to reconstruct input images with the addition of extra noise. Language models also use self-supervision to make predictions about future words given a sequence of words as input. Such examples demonstrate how we can still learn useful information from large datasets even with no labels, which we then generalize to labeled datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TnK33s6fqz0d"
      },
      "source": [
        "## Section 5.2: Continual Learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vbg86CUPNrt2"
      },
      "source": [
        "Another desirable characteristic for deep learning models is the ability to keep learning as they encounter increasing amounts of different problems. This is especially relevant for neural networks given that they tend to suffer from catastrophic forgetting, which means they forget the results of previous observations as newer observations come in. In gradient descent, neural networks are expected to forget the earliest training examples as optimization proceeds with incoming new observations.\n",
        "\n",
        "Continual learning aims to resolve this issue of neural networks through several techniques. One potential strategy involves adding increasingly more neurons as a model learns. The new neurons will be paired with newer examples, while older neurons are still kept to retain information learned from previous examples. Another strategy involves specialized regularization designed to prevent too much change from occurring in the weights of the model. Soft penalty schemes like Learning without Forgetting (LwF) and elastic weight consolidation (EWC) focus on the adjustment of weights in such a manner that previous examples are not forgotten. \n",
        "\n",
        "Other techniques involve the reintroduction of previous examples into training together with newer examples. Generative replay has the model retain previously acquired knowledge via concurrent replay of artificially generated data meant to resemble previous training examples. Similar methods will pull in previous examples and intersperse them with newer examples as training progresses. Curriculum learning describes a framework in which a model first starts out training on easy examples of a task and then gradually move onto examples of increasing task difficulty. This takes advantage of previous examples by iteratively training on parts of a dataset in stages, building upon previous knowledge."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M34zURwnq2W_"
      },
      "source": [
        "## Section 5.3: Curiosity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ko9YC1v3Toff"
      },
      "source": [
        "Despite the attempts at reasoning and understanding that self-supervision and continual learning embody, these techniques alone are often not enough if want deep learning models to convincingly mimic the thought processes and behavior of human beings. Unlike humans, most deep learning models do not incorporate exploration and reasoning into their learning processes.\n",
        "\n",
        "This has led to the research question of asking what can be done to make AI and deep learning models more curious in learning more about the world around them by exploring. Reinforcement learning systems incorporate exploration with the use of reward functions designed to motivate them to move to a state or take an action which is surprising or unexpected. Combined with a usual reward function, these specialized reward functions are meant to encourage exploration by the RL system in addition to exploitation.\n",
        "\n",
        "Additional strategies have tried adding onto this idea of exploration. Such methods include the use of episodic memory to have RL systems remember what has been seen before and reward functions which encourage RL systems to discover novel environments. These are meant to help RL systems not only take into account rules and constraints but also previous observations when making decisions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBpRge20HJZm"
      },
      "source": [
        "Children spend a vast amount of time “playing” -- just picking things up and putting them in their mouths. What might they be learning?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "dVF8vn-lHJZm"
      },
      "source": [
        "children_learning = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nzYwETGLHJ3W"
      },
      "source": [
        "What would be a simple loss function that they (or a child-like robot) might be optimizing on?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "Ik8dWjisHJ3X"
      },
      "source": [
        "child_loss_function = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6EsNJ5OHh6ua"
      },
      "source": [
        "*Estimated time: 130 minutes since start*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbgTqgB5rH_W"
      },
      "source": [
        "---\n",
        "# Wrap up\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJ61Rzja3QFR",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Conclusion\n",
        "\n",
        "try: t6;\n",
        "except NameError: t6=time.time()\n",
        "\n",
        "video = YouTubeVideo(id=\"CrA9z0mlmzU\", width=854, height=480, fs=1)\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\n",
        "\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "98YXpSntrY6k",
        "cellView": "form"
      },
      "source": [
        "#@markdown #Run Cell to Show Airtable Form\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\n",
        "\n",
        "import time\n",
        "import numpy as np\n",
        "import urllib.parse\n",
        "from IPython.display import IFrame\n",
        "def prefill_form(src, fields: dict):\n",
        "  '''\n",
        "  src: the original src url to embed the form\n",
        "  fields: a dictionary of field:value pairs,\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\n",
        "  '''\n",
        "  prefill_fields = {}\n",
        "  for key in fields:\n",
        "      new_key = 'prefill_' + key\n",
        "      prefill_fields[new_key] = fields[key]\n",
        "  prefills = urllib.parse.urlencode(prefill_fields)\n",
        "  src = src + prefills\n",
        "  return src\n",
        "\n",
        "\n",
        "#autofill time if it is not present\n",
        "try: t0;\n",
        "except NameError: t0 = time.time()\n",
        "try: t1;\n",
        "except NameError: t1 = time.time()\n",
        "try: t2;\n",
        "except NameError: t2 = time.time()\n",
        "try: t3;\n",
        "except NameError: t3 = time.time()\n",
        "try: t4;\n",
        "except NameError: t4 = time.time()\n",
        "try: t5;\n",
        "except NameError: t5 = time.time()\n",
        "try: t6;\n",
        "except NameError: t6 = time.time()\n",
        "\n",
        "#autofill fields if they are not present\n",
        "#a missing pennkey and pod will result in an Airtable warning\n",
        "#which is easily fixed user-side.\n",
        "try: my_pennkey;\n",
        "except NameError: my_pennkey = \"\"\n",
        "try: my_pod;\n",
        "except NameError: my_pod = \"Select\"\n",
        "try: learning_from_the_course;\n",
        "except NameError: learning_from_the_course = \"\"\n",
        "try: deep_learning_limits;\n",
        "except NameError: deep_learning_limits = \"\"\n",
        "try: one_second_rule;\n",
        "except NameError: one_second_rule = \"\"\n",
        "try: invariance_examples;\n",
        "except NameError: invariance_examples = \"\"\n",
        "try: why_no_rules;\n",
        "except NameError: why_no_rules = \"\"\n",
        "try: neurosymbolic_learning;\n",
        "except NameError: neurosymbolic_learning = \"\"\n",
        "try: modular_causality;\n",
        "except NameError: modular_causality = \"\"\n",
        "try: simulation_signature;\n",
        "except NameError: simulation_signature = \"\"\n",
        "try: learning_simulations;\n",
        "except NameError: learning_simulations = \"\"\n",
        "try: children_learning;\n",
        "except NameError: children_learning = \"\"\n",
        "try: child_loss_function;\n",
        "except NameError: child_loss_function = \"\"\n",
        "\n",
        "times = np.array([t1,t2,t3,t4,t5,t6])-t0\n",
        "\n",
        "fields = {\n",
        "    \"pennkey\": my_pennkey,\n",
        "    \"pod\": my_pod,\n",
        "    \"learning_from_the_course\": learning_from_the_course,\n",
        "    \"deep_learning_limits\": deep_learning_limits,\n",
        "    \"one_second_rule\": one_second_rule,\n",
        "    \"invariance_examples\": invariance_examples,\n",
        "    \"why_no_rules\": why_no_rules,\n",
        "    \"neurosymbolic_learning\": neurosymbolic_learning,\n",
        "    \"modular_causality\": modular_causality,\n",
        "    \"simulation_signature\": simulation_signature,\n",
        "    \"learning_simulations\": learning_simulations,\n",
        "    \"children_learning\": children_learning,\n",
        "    \"child_loss_function\": child_loss_function,\n",
        "    \"cumulative_times\": times\n",
        "}\n",
        "\n",
        "src = \"https://airtable.com/embed/shr9LNO1GZueuR3U9?\"\n",
        "\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8vMoA3JGqmW"
      },
      "source": [
        "## Feedback\n",
        "How could this session have been better? How happy are you in your group? How do you feel right now?\n",
        "\n",
        "Feel free to use the embeded form below or use this link:\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrNSJ5ECXhNhsYss\">https://airtable.com/shrNSJ5ECXhNhsYss</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a7xLwg7OGrIe"
      },
      "source": [
        "display(IFrame(src=\"https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red\", width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}